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ABSTRACT 



California has developed a new series of credential 
examinations for foreign language teachers to make it possible to test the 
competence of teachers in six languages less frequently taught than Spanish 
and French. These languages are German, Japanese, Mandarin, Punjabi, Russian, 
and Vietnamese. Each assessment is based on the specifications designed for 
previously developed tests for teachers of Spanish and French, and each 
evaluated cultural and grammatical knowledge. California foreign language 
educators played a significant role in the development of the six 
assessments- The proposed specifications called for assessing proficiency in 
speaking, listening, reading, and writing and knowledge of general 
linguistics concepts and the target culture. Proposed specifications were 
reviewed and test items were developed. The reviewed items were field tested, 
and test forms were prepared after holistic scoring of field tests. 

Additional content reviews were then undertaken to prepare final versions 
that provide comprehensive assessment of teacher knowledge and skills. An 
appendix presents general test specifications for the six tests. (SLD) 
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Innovations in Foreign Language Educator Assessment in California 



Introduction 

Over the past twenty years, California has become not only the most populous state in the nation, but also the most 
diverse. Drawn by its natural beauty, its climate, and its economic opportunities, millions of immigrants from Asia, 
Europe, and Latin America have transformed California into a multiethnic, multicultural, and multilingual society. One 
result of this profound demographic transformation is the substantial proportion of students in California public schools 
whose primary language is other than English, a phenomenon that has led to the development of wide-ranging English-as- 
a-Second-Language and bilingual education programs. Another result is the variety of foreign language instruction that is 
offered in California schools. 

Given its linguistic diversity, the breadth of foreign language instruction that is part of the public school curriculum, and 
the state's co mmi tment to teaching excellence, California is in a unique position to explore new ways of measuring 
whether prospective teachers have the knowledge and skills necessary to function effectively in foreign language 
classrooms. The new series of credential examinations for foreign language teachers developed by the California 
Co mmi ssion on Teacher Certification have provided a comprehensive and integrated solution to this challenge. 



Program Context 

Teachers seeking the California single subjects teaching credential for foreign languages may obtain this credential in 
essentially one of two ways: by completing an approved teacher preparation program in the language of emphasis at a 
California college or university; or by taking and passing a subject matter examination in the language of emphasis. While 
the number of approved California teacher preparation programs in the more commonly taught languages such as Spanish 
and French may be relatively large, there are few, if any, approved teacher preparation programs in the less commonly 
taught languages such as German, Japanese, Mandarin, Punjabi, Russian, and Vietnamese. Given this situation, the 
California Commission on Teacher Credentialing (CCTC), contracted with National Evaluation Systems, Inc. (NES®) to 
develop and administer subject matter credential examinations for foreign language teachers for the six previously 
identified languages, as part of the new Single Subject Assessments for Teaching (SSAT™) program. 

Three years before NES's involvement in the development of these six assessments, the CCTC, working with a panel of 
California public school language teachers and foreign language teacher preparation faculty, had prepared a set of test 
specifications for teachers of languages other than English. These test specifications were designed to assess candidates 
seeking a credential to teach French or Spanish. They were also written from the perspective of a complex assessment 
paradigm that called for three separate tests including a multiple-choice examination of linguistics, literature, 
comprehension of written and spoken language, and knowledge of culture; and two separate performance assessments for 
assessing speaking skills and linguistic, literary, and cultural analysis. 

As part of the changes in state and national best practices in foreign language teacher preparation that emerged after the 
development of these earlier test specifications, the CCTC had received a clear message in its interactions with California 
foreign language educators that a single credential examination that combined all of the relevant knowledge and skill 
domains would be preferable to the three-examination model. A single examination would facilitate candidate registration 
and also reduce total examinee fees, since the entire assessment requirement could be completed in a single test session. 
Also, these same educators called for assessments that reflected the intent of the recently released Final Report of the 
National Standards in Foreign Language Education Project , with its substantial proficiency-based emphasis. Thus, NES 
worked with the CCTC to develop foreign language credentialing examinations in German, Japanese, Mandarin, Punjabi, 
Russian, and Vietnamese that were customized to meet the requests of California foreign language teachers and teacher 
educators. 



CCTC Requirements for Assessments 

Based upon the input of California foreign language educators, the CCTC called for six foreign language subject matter 
assessments. Each assessment would: 

• be based on modified foreign language test specifications (slightly different from the original test specifications 
developed for French and Spanish) that could be applied to varied and multiple languages; 

• provide significant evaluation of both cultural and grammatical knowledge; 
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• provide for ample opportunities to demonstrate language proficiency; 

• identify the most salient traditional and contemporary cultural knowledge needed by entiy-level teachers of each 
language; 

• create authentic tasks that were culturally and linguistically appropriate for diverse languages; and 

• provide for performance-based listening and reading assessments. 



Test Development Process 

California educator involvement. California foreign language educators played a substantial role in the development of 
the six foreign language teacher credential assessments. With the assistance of the CCTC and the California Foreign 
Language Teachers Association (CFLTA), assessment advisoiy panels of up to eight California public school teachers and 
college teacher preparation faculty were appointed for each language. Panelists were all experienced educators and had 
been identified as being active in state and/or national foreign language education activities. The CFLTA provided 
valuable input on California and national trends in foreign language instruction, particularly on some of the unique issues 
of some of the less commonly taught languages. 

Reference materials and ancillary services. Working with the CCTC, National Evaluation Systems assembled an 
extensive library of foreign language instruction and assessment materials, including a number of curriculum documents 
from the California Department of Education, the CFLTA, the American Council on the Teaching of Foreign Languages 
(ACTFL), and the National Standards in Foreign Languages Education Report. 

Development of test specifications. The development of test specifications for the six languages was an iterative process 
that drew on the previously developed and CCTC-approved French and Spanish test specifications, but also took into 
account the special challenges posed by these six diverse languages. 

This process involved review of state and national foreign language standards and curricula, as well as extensive 
consultation with foreign language educators in California. Drawing on these sources, test development staff first prepared 
preliminary outlines of test domains, including examples of content and skills that would be measured within those 
domains. Following review of this draft test framework by language education experts, test development specialists 
prepared detailed test specifications featuring a description of assessment components, sample selected- and constructed- 
response items, and sample performance characteristics on which scoring scales would ultimately be constructed. 

The proposed test specifications called for assessing proficiency in the communicative skills of speaking, listening, 
reading, and writing; knowledge of general linguistics concepts, including the process of language acquisition and 
knowledge of the linguistics of the target language; and knowledge of the target culture. These proficiency and content- 
knowledge domains were expanded into major topics under two broad headings: Knowledge of Language and Culture and 
Language Proficiency . A complete set of the test specifications is provided in Appendix A. 

Assessment methods. Among the proposed assessment methods were the following: 

• selected-response items to assess knowledge of culture, general linguistics, and target-language-specific linguistics; 

• a performance-based language structures task that requires the candidate to complete, transform, and correct sentences 
in the target language; 

• a performance-based listening comprehension assessment in response to target language speech, one recorded sample 
of informal target language (e.g., a brief dialogue among friends), and one recorded sample of relatively formal target 
language speech (e.g., an excerpt from a lecture or radio broadcast); 

• a performance-based reading comprehension assessment of two different passages, one sample from a work of 
literature, the other from a nonliterary source; 

• a performance-based written expression assessment, in which the candidate produces two well-organized compositions 
in response to prompts describing different writing tasks (e.g., a memorandum, a letter, a short essay); and 

• a performance-based oral expression assessment, in which two speaking prompts are presented in written form and are 
also read aloud on tape, with the candidate required to speak on tape in the target language for up to two minutes for 
each prompt. 
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Correlation study. When the test specifications were close to final, they were correlated with the California Standards of 
Quality and Effectiveness for Teacher Preparation in Languages Other than English, a set of standards for teacher 
preparation of language teachers that had been drafted by California foreign language educators. The results of this 
correlation study, which were confirmed by California foreign language educator reviewers, suggested that the proposed 
test specifications and assessment descriptions provided substantial coverage for all of the nine Standards of Quality and 
Effectiveness. 

Test specifications review conference. The assessment advisory panels representing the six languages were convened for 
a two-day meeting to review the draft test specifications and sample test items for each assessment domain. Through a 
series of cross-language general orientation and training sessions combined with review and discussions of issues unique to 
each language, a set of test specifications was approved by all six assessment advisory panels. This set of test 
specifications was supplemented by language-specific notes, through which each language panel provided special 
instructions to the CCTC and to the assessment contractor (e.g., the recommended age, gender, and dialect of recorded 
speakers on the audio tapes for the listening comprehension section of the test; the use of diacritical marks for written 
materials; and the preferred typeface for written materials in the target language). 

Test item development. Following the test specifications review conference, the draft test items and performance 
assessments were developed to match the California-approved test specifications. Throughout the item development 
process, members of the six California assessment advisory panels were contacted to obtain their feedback on draft test 
materials and their suggestions for enhancing those materials. After draft item banks were prepared, the banks were 
reviewed by the contractor's Equity Advisory Board, a group of educators who have experience with issues of bias and 
diversity. The banks were then reviewed by content experts in each of the six languages who had not been previously 
involved in item development activities. 

Item review conference. The six assessment advisory panels were then reconvened for a two-day item review conference. 
Each selected-response and performance assessment was reviewed. If necessary, items were revised, or in some cases 
deleted, based on the consensus decisions of the panels. Following this conference, the banks were put into final form 
based on the panel revisions. 

Field testing. Each of the panel-approved performance assessments was field tested with teacher preparation candidates 
who were judged to possess at least moderate target language proficiency (e.g., raised in a household in which the target 
language was spoken on a regular basis, had spent at least six months living in the target language culture, had completed 
an academic degree or a minor in the target language). 

The field test responses were scored by California educators who were public school or college teachers of the respective 
target language. Three to four scorers trained in the methods of focused holistic scoring participated in the scoring of each 
language. A four-point holistic scoring scale was used to score the field test responses. Across the six languages there was 
high scorer agreement on the holistic ratings assigned to each written and oral response. Marker responses for each 
language— a set of responses clearly demonstrating the holistic score points of "1", "2", "3", and "4"— were identified by 
the scorers for each performance assessment. These would be used to assist in training scorers following future operational 
test administrations. 

Item validation/standard setting. Test forms were prepared following the completion of holistic scoring of field test 
responses. Test forms were prepared for each of the six language fields according to test blueprints that were approved by 
the CCTC and by representatives of each language advisory panel. 

Every test form consisted of written and audio portions, each component raising unique issues and requiring its own 
creative solutions. Large blocks of language text that formed the basis of several types of constructed-response 
assessments were input directly into an electronic test bank. Meanwhile, language-specific "fragments"— individual words, 
phrases, or sentences — were computer scanned and electronically merged into the appropriate selected-response item. 
Simultaneously, NES recruited two native speakers in each target language for the listening sections' audiotaped stimuli. 
The speakers reviewed the committee-approved scripts prior to recording, then worked in pairs to address and resolve any 
issues that arose during the recording session. Once the written and audio portions of the test banks were complete, 
individual test forms were created and reviewed once more by language experts to assure that no error had been introduced 
during the test bank conversion process. 
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Once test forms and ancillary materials had been produced, assessment advisory panel members for all six languages were 
convened for an item validation/standard setting conference. During this meeting the following activities were completed: 

• Each item in the test item bank was independently validated for its match with the intended test specification, 
accuracy, freedom from bias, and job-relatedness. 

• Using both individual and group iterative procedures, minimum passing scores were recommended for each of the 
tests. 

• The holistic scoring scale for each type of performance assessment was reviewed and approved by each panel. 

• Marker responses, obtained from the field test for each of the five holistically scored performance assessments, were 
reviewed and approved. 

The CCTC later approved all of the panel- recommended minimum passing scores. 

Prepare for and administer tests at first operational administration. Following the item validation/standard setting 
conference, any panel-identified editorial revisions were made to the test forms, additional content expert reviews were 
undertaken, and the test forms and audiotapes were copied. Following the first test administration, multiple-choice item 
statistics were reviewed. Concurrently, the performance assessments were scored using focused holistic scoring 
procedures. The multiple-choice and performance assessment domain scores for individual examinees were then combined 
based on CCTC-approved weighting formulas, and score reports were distributed to examinees, teacher preparation 
colleges and universities, and to the CCTC. 



Summary 

At the end of the twentieth century, California is leading the way toward the multicultural society of the future. Faced 
with the challenge of developing a comprehensive series of assessments to effectively measure the knowledge and skills of 
prospective foreign language teachers in multiple target languages, the California Commission on Teacher Credentialing 
and its assessment contractor, National Evaluation Systems, created rigorous examinations utilizing sophisticated selected- 
response items and authentic, performance-based tasks in an integrated assessment of linguistic and grammatical 
knowledge, cultural understanding, and language proficiency. The complex, iterative test development process drew on 
the most contemporary formulations of foreign language instruction standards and relied on the expertise of California 
foreign language educators at each stage of test development, administration, and scoring. The result was a comprehensive 
assessment of prospective foreign language teachers grounded in the necessary knowledge and skills identified by 
California educators, sensitive to the cultural and linguistic characteristics of each target language, yet adaptable for 
effective use with multiple languages. 
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APPENDIX A 
Test Specifications 



SINGLE SUBJECT ASSESSMENTS FOR TEACHING 
TEST SPECIFICATIONS 

Field 19: Mandarin 
Field 20: German 
Field 21: Japanese 
Field 22: Russian 
Field 23: Punjabi 
Field 24: Vietnamese 



CULTURAL KNOWLEDGE 

In this section of the assessment, the candidate responds to multiple-choice questions that may address, but are not limited 
to, the following topics. 

• Physical and cultural geography. 

• Contemporary and historical social structures, and cultural issues, including, but not limited to: political, religious, 
and economic systems and institutions; social classes; education; family and kinship; social customs; work and leisure 
patterns; language. 

• Classical and contemporary literary works and oral traditions. 

• Individual variations within the cultures represented by the people who speak the language. 

• Classical and folk arts. 

• Crosscultural comparisons. 

LINGUISTICS 

In this section of the assessment, the candidate responds to multiple-choice questions that address both knowledge of 
general linguistics and language-specific linguistics. 

A. General Linguistics 

This section of the assessment contains multiple-choice questions that require the candidate to demonstrate his/her 
knowledge of the nature of language, its acquisition, and its use. Items may address, but are not limited to, the 
following topics. 

• Nature of language, its purposes, uses and misuses, and relation to ways of life. 

• The significance of language changes and variations that occur within the contexts of time, place, age, gender, 
and situation. 

• Theories of language acquisition and learning. 

• The co mmuni cation process, with the use of strategies such as pragmatics, discourse analysis, and turn-taking and 
other conversation conventions; proxemics and kinesis; idioms and humor; debate and negotiations; as well as 
suasive and hortatory modes. 

B. Language-Specific Linguistics 

This section of the assessment contains multiple-choice questions that require the candidate to demonstrate his/her 
knowledge of linguistic components of the target language. Items may address, but are not limited to, the following 
topics. 

• Basic linguistic and paralinguistic components of the language: phonology, morphology, syntax, lexicon, 

semantics, suprasegmentals, kinesis, proxemics, grammar and words (meaning and usage). 

• Use of language components in authentic communication contexts. 

• Error analysis and its role in the language learning process. 
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• Principles of contrastive analysis. 

• Dialectal differences, their origins, social implications, and ways to address these differences positively. 
LANGUAGE STRUCTURES ASSESSMENT 

In this section of the assessment, the candidate is presented with scorable tasks that require the candidate to complete, 
transform, and correct sentences in the target language. Tasks for each skill are embedded in a sample of connected 
discourse consisting of one or more sentences. The candidate is required to apply knowledge of grammar, syntax, and 
lexicon to achieve clarity, accuracy, and appropriateness of expression. 

LISTENING ASSESSMENT 

The candidate is presented with two performance assessments, each based on a recorded sample of speech. One sample is 
relatively informal (e.g., a brief dialogue overheard on the street), the other relatively formal (e.g., an excerpt from a 
lecture or radio broadcast). For each sample, the candidate is asked a question testing literal and inferential comprehension 
of the recorded material, then responds in writing to a series of interrelated questions. The candidate needs to demonstrate 
the ability to make auditory discriminations (e.g., verb tenses), recognize vocabulary, understand content, and display 
cultural awareness. 

READING ASSESSMENT 

The candidate is presented with two performance assessments, each based on a passage of text. One sample is from a work 
of literature, the other from a nonliterary source (e.g., a newspaper story, a magazine article, a personal letter). For each 
text, candidates write an answer to a question testing literal and inferential comprehension of the material. The question 
on the literary excerpt may address cultural characteristics and references, style, and use of literary devices. 

WRITING ASSESSMENT 

The candidate is required to produce two well-organized compositions in the target language in response to two prompts 
describing different writing tasks (e.g., a memorandum, a letter, a short essay). The candidate needs to demonstrate the 
ability to communicate in an appropriate register, using a range of vocabulary, idiomatic expressions, and simple and 
complex linguistic structures, and to display an appropriate level of cultural knowledge and sensitivity. 

SPEAKING ASSESSMENT 

Two prompts, presented in written form in the test booklet and also read aloud on tape, require the candidate to speak on 
tape for up to two minutes in the target language. Both prompts represent tasks classified as advanced according to 
ACTFL standards (e.g., narrating in past time, hypothesizing on a personal topic). The candidate needs to demonstrate the 
ability to communicate effectively, using an appropriate range of vocabulary, idiomatic expressions, and simple and 
complex language structures, displaying an appropriate level of cultural knowledge and understanding. 
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